Online speech synthesis using a chronically implanted brain-computer interface in an individual with ALS.

Brain-computer interfaces (BCIs) that reconstruct and synthesize speech using brain activity recorded with intracranial electrodes may pave the way toward novel communication interfaces for people who have lost their ability to speak, or who are at high risk of losing this ability, due to neurological disorders. Here, we report online synthesis of intelligible words using a chronically implanted brain-computer interface (BCI) in a man with impaired articulation due to ALS, participating in a clinical trial (ClinicalTrials.gov, NCT03567213) exploring different strategies for BCI communication. The 3-stage approach reported here relies on recurrent neural networks to identify, decode and synthesize speech from electrocorticographic (ECoG) signals acquired across motor, premotor and somatosensory cortices. We demonstrate a reliable BCI that synthesizes commands freely chosen and spoken by the participant from a vocabulary of 6 keywords previously used for decoding commands to control a communication board. Evaluation of the intelligibility of the synthesized speech indicates that 80% of the words can be correctly recognized by human listeners. Our results show that a speech-impaired individual with ALS can use a chronically implanted BCI to reliably produce synthesized words while preserving the participant's voice profile, and provide further evidence for the stability of ECoG for speech-based BCIs.

Stable Decoding from a Speech BCI Enables Control for an Individual with ALS without Recalibration for 3 Months.

Brain-computer interfaces (BCIs) can be used to control assistive devices by patients with neurological disorders like amyotrophic lateral sclerosis (ALS) that limit speech and movement. For assistive control, it is desirable for BCI systems to be accurate and reliable, preferably with minimal setup time. In this study, a participant with severe dysarthria due to ALS operates computer applications with six intuitive speech commands via a chronic electrocorticographic (ECoG) implant over the ventral sensorimotor cortex. Speech commands are accurately detected and decoded (median accuracy: 90.59%) throughout a 3-month study period without model retraining or recalibration. Use of the BCI does not require exogenous timing cues, enabling the participant to issue self-paced commands at will. These results demonstrate that a chronically implanted ECoG-based speech BCI can reliably control assistive devices over long time periods with only initial model training and calibration, supporting the feasibility of unassisted home use.

Online speech synthesis using a chronically implanted brain-computer interface in an individual with ALS.

Recent studies have shown that speech can be reconstructed and synthesized using only brain activity recorded with intracranial electrodes, but until now this has only been done using retrospective analyses of recordings from able-bodied patients temporarily implanted with electrodes for epilepsy surgery. Here, we report online synthesis of intelligible words using a chronically implanted brain-computer interface (BCI) in a clinical trial participant (ClinicalTrials.gov, NCT03567213) with dysarthria due to amyotrophic lateral sclerosis (ALS). We demonstrate a reliable BCI that synthesizes commands freely chosen and spoken by the user from a vocabulary of 6 keywords originally designed to allow intuitive selection of items on a communication board. Our results show for the first time that a speech-impaired individual with ALS can use a chronically implanted BCI to reliably produce synthesized words that are intelligible to human listeners while preserving the participants voice profile.

Shared Control of Bimanual Robotic Limbs With a Brain-Machine Interface for Self-Feeding.

Advances in intelligent robotic systems and brain-machine interfaces (BMI) have helped restore functionality and independence to individuals living with sensorimotor deficits; however, tasks requiring bimanual coordination and fine manipulation continue to remain unsolved given the technical complexity of controlling multiple degrees of freedom (DOF) across multiple limbs in a coordinated way through a user input. To address this challenge, we implemented a collaborative shared control strategy to manipulate and coordinate two Modular Prosthetic Limbs (MPL) for performing a bimanual self-feeding task. A human participant with microelectrode arrays in sensorimotor brain regions provided commands to both MPLs to perform the self-feeding task, which included bimanual cutting. Motor commands were decoded from bilateral neural signals to control up to two DOFs on each MPL at a time. The shared control strategy enabled the participant to map his four-DOF control inputs, two per hand, to as many as 12 DOFs for specifying robot end effector position and orientation. Using neurally-driven shared control, the participant successfully and simultaneously controlled movements of both robotic limbs to cut and eat food in a complex bimanual self-feeding task. This demonstration of bimanual robotic system control via a BMI in collaboration with intelligent robot behavior has major implications for restoring complex movement behaviors for those living with sensorimotor deficits.

Characteristics and stability of sensorimotor activity driven by isolated-muscle group activation in a human with tetraplegia.

Understanding the cortical representations of movements and their stability can shed light on improved brain-machine interface (BMI) approaches to decode these representations without frequent recalibration. Here, we characterize the spatial organization (somatotopy) and stability of the bilateral sensorimotor map of forearm muscles in an incomplete-high spinal-cord injury study participant implanted bilaterally in the primary motor and sensory cortices with Utah microelectrode arrays (MEAs). We built representation maps by recording bilateral multiunit activity (MUA) and surface electromyography (EMG) as the participant executed voluntary contractions of the extensor carpi radialis (ECR), and attempted motions in the flexor carpi radialis (FCR), which was paralytic. To assess stability, we repeatedly mapped and compared left- and right-wrist-extensor-related activity throughout several sessions, comparing somatotopy of active electrodes, as well as neural signals both at the within-electrode (multiunit) and cross-electrode (network) levels. Wrist motions showed significant activation in motor and sensory cortical electrodes. Within electrodes, firing strength stability diminished as the time increased between consecutive measurements (hours within a session, or days across sessions), with higher stability observed in sensory cortex than in motor, and in the contralateral hemisphere than in the ipsilateral. However, we observed no differences at network level, and no evidence of decoding instabilities for wrist EMG, either across timespans of hours or days, or across recording area. While map stability differs between brain area and hemisphere at multiunit/electrode level, these differences are nullified at ensemble level.

Perceived timing of cutaneous vibration and intracortical microstimulation of human somatosensory cortex.

BACKGROUND: Intracortical microstimulation (ICMS) of somatosensory cortex can partially restore the sense of touch. Though ICMS bypasses much of the neuraxis, prior studies have found that conscious detection of touch elicited by ICMS lags behind the detection of cutaneous vibration. These findings may have been influenced by mismatched stimulus intensities, which can impact temporal perception. OBJECTIVE: Evaluate the relative latency at which intensity-matched vibration and ICMS are perceived by a human participant. METHODS: One person implanted with microelectrode arrays in somatosensory cortex performed reaction time and temporal order judgment (TOJ) tasks. To measure reaction time, the participant reported when he perceived vibration or ICMS. In the TOJ task, vibration and ICMS were sequentially presented and the participant reported which stimulus occurred first. To verify that the participant could distinguish between stimuli, he also performed a modality discrimination task, in which he indicated if he felt vibration, ICMS, or both. RESULTS: When vibration was matched in perceived intensity to high-amplitude ICMS, vibration was perceived, on average, 48 ms faster than ICMS. However, in the TOJ task, both sensations arose at comparable latencies, with points of subjective simultaneity not significantly different from zero. The participant could discriminate between tactile modalities above chance level but was more inclined to report feeling vibration than ICMS. CONCLUSIONS: The latencies of ICMS-evoked percepts are slower than their mechanical counterparts. However, differences in latencies are small, particularly when stimuli are matched for intensity, implying that ICMS-based somatosensory feedback is rapid enough to be effective in neuroprosthetic applications.

Sequential epiretinal stimulation improves discrimination in simple shape discrimination tasks only.

Objective. Electrical stimulation of the retina can elicit flashes of light called phosphenes, which can be used to restore rudimentary vision for people with blindness. Functional sight requires stimulation of multiple electrodes to create patterned vision, but phosphenes tend to merge together in an uninterpretable way. Sequentially stimulating electrodes in human visual cortex has recently demonstrated that shapes could be 'drawn' with better perceptual resolution relative to simultaneous stimulation. The goal of this study was to evaluate if sequential stimulation would also form clearer shapes when the retina is the neural target.Approach. Two human participants with retinitis pigmentosa who had Argus®II epiretinal prostheses participated in this study. We evaluated different temporal parameters for sequential stimulation and performed phosphene shape mapping and forced choice discrimination tasks. For the discrimination tasks, performance was compared between stimulating electrodes simultaneously versus sequentially.Main results. Phosphenes elicited by different electrodes were reported as vastly different shapes. For sequential stimulation, the optimal pulse train duration was 200 ms when stimulating at 20 Hz and the optimal gap interval was tied between 0 and 50 ms. Sequential electrode stimulation outperformed simultaneous stimulation in simple discrimination tasks, in which shapes were created by stimulating 3-4 electrodes, but not in more complex discrimination tasks involving ≥5 electrodes. The efficacy of sequential stimulation depended strongly on selecting electrodes that elicited phosphenes with similar shapes and sizes.Significance. An epiretinal prosthesis can produce coherent simple shapes with a sequential stimulation paradigm, which can be used as rudimentary visual feedback. However, success in creating more complex shapes, such as letters of the alphabet, is still limited. Sequential stimulation may be most beneficial for epiretinal prostheses in simple tasks, such as basic navigation, rather than complex tasks such as novel object identification.

Intracortical Somatosensory Stimulation to Elicit Fingertip Sensations in an Individual With Spinal Cord Injury.

BACKGROUND AND OBJECTIVES: The restoration of touch to fingers and fingertips is critical to achieving dexterous neuroprosthetic control for individuals with sensorimotor dysfunction. However, localized fingertip sensations have not been evoked via intracortical microstimulation (ICMS). METHODS: Using a novel intraoperative mapping approach, we implanted electrode arrays in the finger areas of left and right somatosensory cortex and delivered ICMS over a 2-year period in a human participant with spinal cord injury. RESULTS: Stimulation evoked tactile sensations in 8 fingers, including fingertips, spanning both hands. Evoked percepts followed expected somatotopic arrangements. The subject was able to reliably identify up to 7 finger-specific sites spanning both hands in a finger discrimination task. The size of the evoked percepts was on average 33% larger than a finger pad, as assessed via manual markings of a hand image. The size of the evoked percepts increased modestly with increased stimulation intensity, growing 21% as pulse amplitude increased from 20 to 80 µA. Detection thresholds were estimated on a subset of electrodes, with estimates of 9.2 to 35 µA observed, roughly consistent with prior studies. DISCUSSION: These results suggest that ICMS can enable the delivery of consistent and localized fingertip sensations during object manipulation by neuroprostheses for individuals with somatosensory deficits. CLINICALTRIALSGOV IDENTIFIER: NCT03161067.

Intracortical microstimulation of somatosensory cortex enables object identification through perceived sensations.

Advances in brain-machine interfaces have helped restore function and independence for individuals with sensorimotor deficits; however, providing efficient and effective sensory feedback remains challenging. Intracortical microstimulation (ICMS) of sensorimotor brain regions is a promising technique for providing bioinspired sensory feedback. In a human participant with chronically-implanted microelectrode arrays, we provided ICMS to the primary somatosensory cortex to generate tactile percepts in his hand. In a 3-choice object identification task, the participant identified virtual objects using tactile sensory feedback and no visual information. We evaluated three different stimulation paradigms, each with a different weighting of the grip force and its derivative, to explore the potential benefits of a more bioinspired stimulation strategy. In all paradigms, the participant's ability to identify the objects was above-chance, with object identification accuracy reaching 80% correct when using only sustained grip force feedback and 76.7% when using equal weighting of both sustained grip force and its derivative. These results demonstrate that bioinspired ICMS can provide sensory feedback that is functionally beneficial in sensorimotor tasks. Designing more efficient stimulation paradigms is important because it will allow us to 1) provide safer stimulation delivery methods that reduce overall injected charge without sacrificing function and 2) more effectively transmit sensory information to promote intuitive integration and usage by the human body.

Novel intraoperative online functional mapping of somatosensory finger representations for targeted stimulating electrode placement: technical note.

Defining eloquent cortex intraoperatively, traditionally performed by neurosurgeons to preserve patient function, can now help target electrode implantation for restoring function. Brain-machine interfaces (BMIs) have the potential to restore upper-limb motor control to paralyzed patients but require accurate placement of recording and stimulating electrodes to enable functional control of a prosthetic limb. Beyond motor decoding from recording arrays, precise placement of stimulating electrodes in cortical areas associated with finger and fingertip sensations allows for the delivery of sensory feedback that could improve dexterous control of prosthetic hands. In this study, the authors demonstrated the use of a novel intraoperative online functional mapping (OFM) technique with high-density electrocorticography to localize finger representations in human primary somatosensory cortex. In conjunction with traditional pre- and intraoperative targeting approaches, this technique enabled accurate implantation of stimulating microelectrodes, which was confirmed by postimplantation intracortical stimulation of finger and fingertip sensations. This work demonstrates the utility of intraoperative OFM and will inform future studies of closed-loop BMIs in humans.

Similitude assessment method for comparing PMHS response data from impact loading across multiple test devices.

Biological tissue testing is inherently susceptible to the wide range of variability specimen to specimen. A primary resource for encapsulating this range of variability is the biofidelity response corridor or BRC. In the field of injury biomechanics, BRCs are often used for development and validation of both physical, such as anthropomorphic test devices, and computational models. For the purpose of generating corridors, post-mortem human surrogates were tested across a range of loading conditions relevant to under-body blast events. To sufficiently cover the wide range of input conditions, a relatively small number of tests were performed across a large spread of conditions. The high volume of required testing called for leveraging the capabilities of multiple impact test facilities, all with slight variations in test devices. A method for assessing similitude of responses between test devices was created as a metric for inclusion of a response in the resulting BRC. The goal of this method was to supply a statistically sound, objective method to assess the similitude of an individual response against a set of responses to ensure that the BRC created from the set was affected primarily by biological variability, not anomalies or differences stemming from test devices.

Embedded clutter reduction and face detection algorithms for a visual prosthesis.

Retinal prosthetic devices can significantly and positively impact the ability of visually challenged individuals to live a more independent life. We describe a visual processing system which leverages image analysis techniques to produce visual patterns and allows the user to more effectively perceive their environment. These patterns are used to stimulate a retinal prosthesis to allow self guidance and a higher degree of autonomy for the affected individual. Specifically, we describe an image processing pipeline that allows for object and face localization in cluttered environments as well as various contrast enhancement strategies in the "implanted image." Finally, we describe a real-time implementation and deployment of this system on the Argus II platform. We believe that these advances can significantly improve the effectiveness of the next generation of retinal prostheses.

Behavioral assessment of sensitivity to intracortical microstimulation of primate somatosensory cortex.

Intracortical microstimulation (ICMS) is a powerful tool to investigate the functional role of neural circuits and may provide a means to restore sensation for patients for whom peripheral stimulation is not an option. In a series of psychophysical experiments with nonhuman primates, we investigate how stimulation parameters affect behavioral sensitivity to ICMS. Specifically, we deliver ICMS to primary somatosensory cortex through chronically implanted electrode arrays across a wide range of stimulation regimes. First, we investigate how the detectability of ICMS depends on stimulation parameters, including pulse width, frequency, amplitude, and pulse train duration. Then, we characterize the degree to which ICMS pulse trains that differ in amplitude lead to discriminable percepts across the range of perceptible and safe amplitudes. We also investigate how discriminability of pulse amplitude is modulated by other stimulation parameters-namely, frequency and duration. Perceptual judgments obtained across these various conditions will inform the design of stimulation regimes for neuroscience and neuroengineering applications.

The effects of chronic intracortical microstimulation on neural tissue and fine motor behavior.

OBJECTIVE: One approach to conveying sensory feedback in neuroprostheses is to electrically stimulate sensory neurons in the cortex. For this approach to be viable, it is critical that intracortical microstimulation (ICMS) causes minimal damage to the brain. Here, we investigate the effects of chronic ICMS on the neuronal tissue across a variety of stimulation regimes in non-human primates. We also examine each animal's ability to use their hand--the cortical representation of which is targeted by the ICMS--as a further assay of possible neuronal damage. APPROACH: We implanted electrode arrays in the primary somatosensory cortex of three Rhesus macaques and delivered ICMS four hours per day, five days per week, for six months. Multiple regimes of ICMS were delivered to investigate the effects of stimulation parameters on the tissue and behavior. Parameters included current amplitude (10-100 µA), pulse train duration (1, 5 s), and duty cycle (1/1, 1/3). We then performed a range of histopathological assays on tissue near the tips of both stimulated and unstimulated electrodes to assess the effects of chronic ICMS on the tissue and their dependence on stimulation parameters. MAIN RESULTS: While the implantation and residence of the arrays in the cortical tissue did cause significant damage, chronic ICMS had no detectable additional effect; furthermore, the animals exhibited no impairments in fine motor control. SIGNIFICANCE: Chronic ICMS may be a viable means to convey sensory feedback in neuroprostheses as it does not cause significant damage to the stimulated tissue.

Long-term stability of sensitivity to intracortical microstimulation of somatosensory cortex.

OBJECTIVE: The dexterous manipulation of objects depends heavily on somatosensory signals from the limb. The development of anthropomorphic robotic arms and of algorithms to decode intended movements from neuronal signals has stimulated the need to restore somatosensation for use in upper-limb neuroprostheses. Without touch and proprioception, patients have difficulty controlling prosthetic limbs to a level that justifies the required invasive surgery. Intracortical microstimulation (ICMS) through chronically implanted electrode arrays has the potential to provide rich and intuitive sensory feedback. This approach to sensory restoration requires, however, that the evoked sensations remain stable over time. APPROACH: To investigate the stability of ICMS-evoked sensations, we measured the ability of non-human primates to detect ICMS over experimental sessions that spanned years. MAIN RESULTS: We found that the performance of the animals remained highly stable over time, even when they were tested with electrodes that had experienced extensive stimulation. SIGNIFICANCE: Given the stability of the sensations that it evokes, ICMS may thus be a viable approach for sensory restoration.

Sensitivity to microstimulation of somatosensory cortex distributed over multiple electrodes.

Meaningful and repeatable tactile sensations can be evoked by electrically stimulating primary somatosensory cortex. Intracortical microstimulation (ICMS) may thus be a viable approach to restore the sense of touch in individuals who have lost it, for example tetraplegic patients. One of the potential limitations of this approach, however, is that high levels of current can damage the neuronal tissue if the resulting current densities are too high. The limited range of safe ICMS amplitudes thus limits the dynamic range of ICMS-evoked sensations. One way to get around this limitation would be to distribute the ICMS over multiple electrodes in the hopes of intensifying the resulting percept without increasing the current density experienced by the neuronal tissue. Here, we test whether stimulating through multiple electrodes is a viable solution to increase the dynamic range of ICMS-elicited sensations without increasing the peak current density. To this end, we compare the ability of non-human primates to detect ICMS delivered through one vs. multiple electrodes. We also compare their ability to discriminate pulse trains differing in amplitude when these are delivered through one or more electrodes. We find that increasing the number of electrodes through which ICMS is delivered only has a marginal effect on detectability or discriminability despite the fact that 2-4 times more current is delivered overall. Furthermore, the impact of multielectrode stimulation (or lack thereof) is found whether pulses are delivered synchronously or asynchronously, whether the leading phase of the pulses is cathodic or anodic, and regardless of the spatial configuration of the electrode groups.

The effect of chronic intracortical microstimulation on the electrode-tissue interface.

OBJECTIVE: Somatosensation is critical for effective object manipulation, but current upper limb prostheses do not provide such feedback to the user. For individuals who require use of prosthetic limbs, this lack of feedback transforms a mundane task into one that requires extreme concentration and effort. Although vibrotactile motors and sensory substitution devices can be used to convey gross sensations, a direct neural interface is required to provide detailed and intuitive sensory feedback. The viability of intracortical microstimulation (ICMS) as a method to deliver feedback depends in part on the long-term reliability of implanted electrodes used to deliver the stimulation. The objective of the present study is to investigate the effects of chronic ICMS on the electrode-tissue interface. APPROACH: We stimulate the primary somatosensory cortex of three Rhesus macaques through chronically implanted electrodes for 4 h per day over a period of six months, with different electrodes subjected to different regimes of stimulation. We measure the impedance and voltage excursion as a function of time and of ICMS parameters. We also test the sensorimotor consequences of chronic ICMS by having animals grasp and manipulate small treats. MAIN RESULTS: We show that impedance and voltage excursion both decay with time but stabilize after 10-12 weeks. The magnitude of this decay is dependent on the amplitude of the ICMS and, to a lesser degree, the duration of individual pulse trains. Furthermore, chronic ICMS does not produce any deficits in fine motor control. SIGNIFICANCE: The results suggest that chronic ICMS has only a minor effect on the electrode-tissue interface and may thus be a viable means to convey sensory feedback in neuroprosthetics.

Restoring the sense of touch with a prosthetic hand through a brain interface.

Our ability to manipulate objects dexterously relies fundamentally on sensory signals originating from the hand. To restore motor function with upper-limb neuroprostheses requires that somatosensory feedback be provided to the tetraplegic patient or amputee. Given the complexity of state-of-the-art prosthetic limbs and, thus, the huge state space they can traverse, it is desirable to minimize the need for the patient to learn associations between events impinging on the limb and arbitrary sensations. Accordingly, we have developed approaches to intuitively convey sensory information that is critical for object manipulation--information about contact location, pressure, and timing--through intracortical microstimulation of primary somatosensory cortex. In experiments with nonhuman primates, we show that we can elicit percepts that are projected to a localized patch of skin and that track the pressure exerted on the skin. In a real-time application, we demonstrate that animals can perform a tactile discrimination task equally well whether mechanical stimuli are delivered to their native fingers or to a prosthetic one. Finally, we propose that the timing of contact events can be signaled through phasic intracortical microstimulation at the onset and offset of object contact that mimics the ubiquitous on and off responses observed in primary somatosensory cortex to complement slowly varying pressure-related feedback. We anticipate that the proposed biomimetic feedback will considerably increase the dexterity and embodiment of upper-limb neuroprostheses and will constitute an important step in restoring touch to individuals who have lost it.

Decoding of individuated finger movements using surface electromyography.

Upper limb prostheses are increasingly resembling the limbs they seek to replace in both form and functionality, including the design and development of multifingered hands and wrists. Hence, it becomes necessary to control large numbers of degrees of freedom (DOFs), required for individuated finger movements, preferably using noninvasive signals. While existing control paradigms are typically used to drive a single-DOF hook-based configurations, dexterous tasks such as individual finger movements would require more elaborate control schemes. We show that it is possible to decode individual flexion and extension movements of each finger (ten movements) with greater than 90% accuracy in a transradial amputee using only noninvasive surface myoelectric signals. Further, comparison of decoding accuracy from a transradial amputee and able-bodied subjects shows no statistically significant difference ( p < 0.05) between these subjects. These results are encouraging for the development of real-time control strategies based on the surface myoelectric signal to control dexterous prosthetic hands.